Time of Flight (ToF) cameras are active range sensors that provide depth images at high frame-rates. They are equipped with a light source, normally infrared, that illuminates an object, for example a human body, or surface which may comprise different materials, wherein the object or surface may be part of a scene, and a CMOS/CCD (Charge-Coupled Device) sensor, as in a standard camera, that captures the reflected infrared light. The distance to the objects in the observed scene is measured based on the time of flight principle, that is, the distance is proportional to the time spent by the signal to reach the surface of an object and come back.
Depth measurements may in this way obtained for each pixel of the CMOS/CCD sensor and be used to produce a depth image. Fast acquisition of depth images is of great use in a wide range of applications, for instance, in robotics, human machine interaction and scene modelling. However, current commercially available devices have a low resolution and are affected by several sources of errors such as noise, systematic wiggling error, reflectivity/integration time error and flying pixels.
Several approaches exist that aim to solve the problem of the improvement of the depth measurements, including different ways to calibrate the ToF camera, fusing ToF camera with single or stereo RGB cameras, or fusing a sequence of depth image for a higher resolutions. There are also a number of methods that combine several ToF in order to create 3D reconstructions. The drawback of such approaches is that they mostly rely at some level on putting in correspondence point clouds derived from the depth images.
A known technique to enhance depth ToF images include calibrating the depth by fitting a non-linear correction function that relates the uncorrected depth, intensity and amplitude. A further approach involves intensity and wiggling adjustment before correcting the depth information. It is also possible to compensate the internal and environmental factor, like the inner temperature, integration time, ambient temperature, light or object properties. Another type of calibration can be made using special reflective checkerboards. All of these approaches involve a large amount of laborious calibration.
A second trend to improve the ToF depth images is to use ToF simultaneously with other cameras. A high-resolution colour camera together with a ToF camera in a calibrated setup allows removing outliers, smoothing the depth images and increasing the depth resolution. Multiple view systems combining several ToF and high-resolution colour cameras have also been used to create 3D textured reconstructions. This clearly increases the costs and complexity of the image acquisition system overall.
Most methods that combine purely ToF depth images rely on finding correspondences between the point clouds generated from different views, e.g. using Iterative Closest Point method. A second option is to combine the depth images in time assuming static scene in order to obtain depth super-resolution. These methods are conducted posterior to the image acquisition and are also complex, involving a large amount of processing.
Finally, a related optimisation method optimises the surface observed by a single ToF camera using shading constraints and photometric properties of the surface with accuracy improvements. However, the accuracy is improved at the significant cost of generally slow optimisations.